1. Field of the Invention
The present invention relates to picture transmission apparatus for transmitting pictures extracted from a picture source, a picture transmission method, a picture transmission program for causing a computer to act as said picture transmission apparatus, and a recording medium on which the program is recorded, and in particular to picture transmission apparatus for transmitting the latest picture data in real time depending on the situation of the network bandwidth, a picture transmission method and a recording medium, and a picture transmission program.
2. Description of the Related Art
With the development of the network technologies in recent years, a network picture transmission system has been in widespread use for transmitting pictures via a network and playing back the picture received by the receiving party. In general, the volume of picture data is huge. Thus, in order to transmit pictures via a network, a wide network bandwidth for transmitting pictures must be reserved. Some of the recent technologies reduces a delay in the playback of pictures even in a network where a certain bandwidth is not guaranteed.
A technology for reducing a delay in the playback time caused by insufficient playback capability on the picture playback side or insufficient network bandwidth in order to maintain the real-time characteristics is disclosed in the Japanese Patent Laid-Open No. 214936/1997. According to this technology, in case reception and decoding of packets takes time on the client that receives pictures, the client discards the packets whose decoding is not complete and receives the latest packets from the server in order to maintain the real-time characteristic.
The network moving picture delivery system (hereinafter referred to as related art picture transmission apparatus) according to the Japanese Patent Laid-Open No. 214936/1997 will be described referring to FIGS. 31 through 37.
As shown in FIG. 31, related art picture transmission apparatus comprises a server 3101 for transmitting pictures, a client 3102 for receiving data and displaying pictures, and a network 3103 for performing data communications between the server 3101 and the client 3102.
The server 3101 comprises a moving picture data storage processor 3111 for extracting picture data from a recording medium such as a hard disk, a server-mounted packet transmission processor 3112 for splitting picture data into packets and transmitting the packets to the client 3102 via the network 3103, and a refreshing data detector 3113 for detecting the first portion of picture data that can be decoded.
The client 3102 comprises a moving picture display processor 3121 for displaying received pictures, a decoding processor 3122 for decoding received picture data and converting the resulting data to a format that can be displayed on a display device such as a display, a packet request/reception processor 3123 for receiving data transmitted from the server 3101 via a network, a decoding delay detector 3124 for calculating the decoding delay time of packets transmitted from the server 3101 based on the time information stored in packets, and a transmission delay detector 3125 for calculating the transmission delay time of packets transmitted from the server 3101 based on the time information owned by packets.
Next, the structure of data the server 3101 transmits to the client 3102 via the network 3103 will be described referring to FIGS. 32 through 34. FIG. 32 is an explanatory drawing that shows a bit string of encoded data composed of inter-frame-encoded data string and refreshing data. FIG. 33 is an explanatory drawing that shows a bit string of encoded data split into a plurality of packets shown in FIG. 32. FIG. 34 is an explanatory drawing that shows the data structure of encoded data with a header added per packet.
When the server 3101 transmits picture data to the client 3102 via the network 3103, the server 3101 uses an encoding system such as H.261 or MPEG that is based on inter-frame difference to encode picture data. In the picture data to be transmitted to the client 3102, picture frames (hereinafter referred to as refreshing data) that is cyclically intra-frame-encoded are cyclically inserted in the inter-frame-encoded data string 3201, as shown in FIG. 32. Encoded data provided via encoding systems such as H.261 and MPEG is represented as a bit string. The encoded data shown in FIG. 33 is split into packets 3301 of an appropriate size in order to facilitate transfer on the network 3103.
Next, as shown in FIG. 34, a header 3402 is added to each packet 3301 split from the encoded data to create packet data 3401. A group of packet data 3401 thus created is held in the server 3101 as a single piece of picture data. The header 3402 of individual packet data 3401 contains a time index 3411 for indicating the transmission time and the encoding end time of the packet data 3401 in the situation where the most recent encoding of bit streams was normally performed, and an identifier 3412 for determining whether the packet contains intra-frame encoded data.
Next, a mechanism of transmitting pictures by the picture transmission apparatus shown in FIG. 31 while reducing the transmission delay or encoding delay will be described referring to FIGS. 35 and 36. FIG. 35 is a conceptual drawing of transmission used in case the transmission time or decoding end time of a single packet is shorter than the transmission time or decoding end time indicated by the time index 3411. FIG. 36 is a conceptual drawing of transmission used in case the transmission time or decoding end time of a single packet is longer than the transmission time or decoding end time indicated by the time index 3411.
The packet request/reception processor 3123 connects to the server-mounted packet transmission processor 3112 via the network 3103 and specifies the contents name of a desired picture. The packet request/reception processor 3123 requests transfer of data in packets from the server-mounted packet transmission processor 3113. The packets received by the packet request/reception processor 3123 are stored in the receiving buffer (not shown) of the packet request/reception processor 3123. The decoding processor 3122 sequentially reads and decodes the packets. On the client 3102, the decoding delay detector 3124 and transmission delay detector 3125 starts measurement of elapsed time of transmission and decoding at the same time the first packet is received from the server-mounted packet transmission processor 3112.
The transmission delay detector 3125 measures the elapsed time from the time the first packet arrived when it receives a new packet. The decoding delay detector 3124 measures the elapsed time from the time the first packet was read from the receiving buffer of the packet request/reception processor 3123 when decoding of picture data contained in the packets received from the packet request/reception processor 3123 is complete. Each elapsed time measured by the transmission delay detector 3125 or decoding delay detector 3124 is compared with the time index 3411 recorded in the header 3402 of each packet.
As shown in FIG. 35, when the elapsed time 3502 measured by the decoding delay detector 3124 or transmission delay detector 3125 is shorter than the decoding end time or transmission time 3501 indicated by the time index 3411, the decoding processor 3122 and the packet request/reception processor 3123 suspends processing until the elapsed time 3502 reaches the time indicated by the time index 3411.
As shown in FIG. 36, when the decoding end time or transmission time 3603, 3604 measured by the decoding delay detector 3124 or transmission delay detector 3125 is longer than the time 3601, 3602, and in case the difference 3611, 3612 between the time 3601, 3602 and the measured time 3603, 3604 is smaller than the allowable value (3611), the decoding processor 3122 and the packet request/reception processor 3123 continues processing. On the other hand, in case the difference 3611, 3612 is larger than the allowable value (3612), the decoding processor suspends processing and performs the following delay reduction processing.
The delay reduction method for reducing a delay used in case the difference such as the difference 3612 is larger than the allowable value will be described referring to FIG. 37. In case the difference is larger than the allowable value as in the difference 3612, the client 3102 suspends the processing of the decoding processor 3122, clears the packets 3711, 3712, 3713, 3714 already received by the packet request/reception processor 3123, then requests refreshing data 3202 emerging after the next transmission-pending packet from the server-mounted packet transmission processor 3112. When requesting refreshing data, the client 3102 also notifies the server 3101 of the value of the difference 3702.
On the server 3101, in response to the refreshing data request by the client 3102, the refreshing data detector 3113 detects the packet 3722 that contains refreshing data emerging after the difference 3702 transmitted to the client 3102 from the time index of the next transmission-scheduled packet, then transmits the packet 3722 and the next packet to the client 3102. On the client 3102, receiving the refreshing data 3202 in response to the request, the packet request/reception processor 3123 requests packets that follows the received packet. The decoding processor 3122 suspends decoding processing until the start time recorded in the packet to eliminate the delay, and restarts decoding processing when the start time is reached.
In this way, according to the related art picture transmission apparatus, when the client 3102 requests transmission of pictures from the server 3101, the server 3101 extracts the desired packets in response to the request and transmits the packets to the client 3102. A well as the aforementioned related art example, there exists an example where the server reads packets from a recording medium such as a hard disk while transmitting packets in order to provide packets to be transmitted next so that packets may be immediately transmitted on receiving a packet request by the client.
The aforementioned picture transmission apparatus encodes the picture data extracted by the server 3101 and transmits the resulting picture data to the client 3102. In case encoding takes time or extraction of picture data takes time, the server 3101 cannot transmit encoded data to the client 3102 from when encoding is started to when it is terminated. Thus the allocated network bandwidth is left unused for some time, so that the frame rate (number of frames that can be transmitted per second) of the pictures received by the client 3102 remains low.
Also, in case the encoded data transmission and picture data encoding are respectively performed in parallel, a wait time is generated from the end of the picture data encoding to the start of encoded data transmission. This wait time causes a delay in the playback of pictures in the client 3102.
When sudden jitter (variation) has occurred in the network, the client 3102 requests discarding of too many frames from the server 3101 as a result of measurement of transmission time. This may cause the frames that can be transmitted to the allocated network bandwidth to be discarded in the server 3101.
When the bandwidth of the network varies, the variation is not followed but the frames ready for transmission are discarded in the server 3101 or packets with a great delay maybe transmitted.
In case the server 3101 extracts picture data before performing encoding, the picture data encoding takes time depending on the load of the central processing unit (CPU) so that the encoded data transmission is not allowed while the CPU is performing picture data encoding. Thus there arises the time an allocated network bandwidth is left unused, and the frame rate of the pictures received by the client 3102 remains low.
Also, in case the encoded data transmission and picture data encoding are respectively performed in parallel, a wait time is generated from the end of the picture data encoding to the start of encoded data transmission. This wait time causes a delay in the playback of pictures in the client 3102.
In case the server 3101 extracts picture data then encodes the extracted picture data, encoding must be performed in a short time in order for the server 3101 to transmit encoded data to the client 3102 without delay in case the client 3102 has requested refreshing data. In such a case, the server 3101 cannot perform high-quality encoding, that is, cannot obtain high-quality compressed data.
When the location where picture data is extracted has changed for example from a hard disk to a vide tape, the server 3101 takes time extracting the packets requested by the client 3102 from the video tape. This causes a delay in the picture playback.
Further, when the server 3101 uses the same network to receive picture data from another server and transmits it to the client 3102, the time required for the server 3101 to receive pictures differs from the time required for the server 3101 to transmit pictures. Thus the server 3101 cannot transmit the received pictures to the client 3102, so that the pictures that cannot be transmitted may be discarded in the server 3101.
Especially, in a monitoring system for performing remote monitoring via a network, pictures showing the latest situation are essential to the monitoring side. However, the network used for transmission of picture data may be subject to fluctuations in the transmission speed. When a camera is directed to a desired direction or zooming-in/out is performed via remote operation from the monitoring side, the picture playback apparatus on the monitoring side does not display desired pictures in the meantime. When in particular the transmission speed drops due to the network fluctuations in addition to this problem, the pictures displayed on the picture playback apparatus on the monitoring side lacks real-time characteristics. In a monitoring system, a delay of a mere one or two seconds is fatal. There has been a need for picture transmission apparatus that can provide the latest and high-accuracy pictures independent of the network situation.
The invention has been proposed in view of the related art problems and providing picture transmission apparatus that can reduce a delay in the picture playback depending on the situation of the network bandwidth when for transmitting pictures via the network, that is, picture transmission apparatus that can transmit the latest picture data in real time, a picture transmission method and a recording medium, and a picture transmission program.